Method of response synthesis in a driver assistance system

ABSTRACT

A method ( 600 ) of synthesizing a response to an operating situation of a vehicle includes the step of generating a master condition list ( 602 ), determining the existence of an operating situation ( 604 ), determining an operator cognitive load ( 606 ) and determining a response to the operating situation based in part on the master condition list and the operator cognitive load ( 608 ). The master condition list is a fusion of sensor data gathered within and around the vehicle. The operator cognitive load is a measure of the operator&#39;s ability to handle tasks within the vehicle.

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims priority to U.S. ProvisionalApplication Serial No. 60/240,443, filed Oct. 13, 2000 entitled “Systemfor Real-Time Driving Performance Assessment;” U.S. ProvisionalApplication Serial No. 60/240,444, filed Oct. 13, 2000 entitled “ASystem for Driver Assistance and Driver Performance Improvement;” U.S.Provisional Application Serial No. 60/240,493, filed Oct. 13, 2000entitled “Driver's Cell Phone Assistant;” U.S. Provisional ApplicationSerial No. 60/240,560, filed Oct. 13, 2000 entitled “Response Selector:A Method of Response Synthesis in Driver Assistance System;” and U.S.Provisional Application Serial No. 60/240,553, filed Oct. 13, 2000entitled “A System for Driver Performance Improvement,” the disclosuresof which are hereby incorporated herein by reference.

FIELD OF THE INVENTION

[0002] The present invention relates generally to the field of vehicleoperation, and more particularly, to a method and apparatus forassessing and improving the performance of a vehicle operator.

BACKGROUND OF THE INVENTION

[0003] The flood of information reaching drivers—telematics,infotainment, collision warning and others—requires a new approach tothe operator-vehicle interface. At present, information (such asnavigation instructions, cell phone and email messages, trafficwarnings, infotainment options, vehicle condition monitoring, etc.) ispresented to the vehicle operator asynchronously taking no account ofhow demanding the driving task might be in a given moment. For example,a “check engine” indicator light might light up among the instruments atthe same time a driver is putting a CD into the stereo system, while thenavigation system screen displays an upcoming turn and gives a verbaldescription of that turn, as a cell phone call comes into the car andthe driver is engaged in conversation with one or more passengers.

[0004] Human beings have a finite ability to perceive the environment,to attend to elements of the environment, to cognitively process thestimuli taken in, to draw appropriate meaning from perceptions, and toact appropriately upon those perceived meanings. Furthermore, there isgreat variation within the driving population in both native anddeveloped abilities to drive. Thus, vehicle operators are subject toconfusion, distraction, and to ignorance, which is exacerbated by thebarrage of stimuli they are now subjected to while operating a vehicle.Training, experience, and technology can be used to mitigate confusion,distraction, and ignorance. Unfortunately, in the United States there islittle formal or informal training in the skills involved in driving,beyond the period when people first apply for their licenses. Drivertraining programs have not proven to be particularly effective, nor istraining continued through the driving career. In fact, in the UnitedStates, in particular, most people think of driving as a right ratherthan a privilege. Further, studies show that most think of themselves asgood drivers and of “the other person” as the one who creates problems.Unless and until a cultural or legal change takes place that encouragesdrivers to wish to improve their driving skill, it seems thattechnological solutions designed to minimize confusion, distraction, andignorance have the best potential for improving the safety of thehighway transportation system, which system is likely to become morecrowded and, with little or no expansion of the roadway infrastructurelikely to occur, therefore, also more dangerous in the future.

[0005] To address these and other safety concerns, an integrated safetysystem based on a state transition model has been proposed. Theunderlying concept is a “hierarchy of threat” model that steps through aseries of states each one representing an assessment of the danger of animpending collision based on information from external object detectorsand in-vehicle sensors. The states are “normal driving state,” “warningstate,” “collision avoidable state,” “collision unavoidable state,”“post-collision state.” Sensor and data fusion algorithms combineinformation from the sensors and determine the degree to which thedanger of collision exists. If the system detects the danger of acollision it issues warnings to the driver or, in some situations, takescontrol of the vehicle and initiates automatic braking, automatic lanechange, or other forms of vehicle control. This system represents anattempt to bring previously unrelated sensor information into anintegrated state from which useful inference about the danger ofcollision may be made and warnings to the driver, or actual control ofthe vehicle, can be used to avoid completely or mitigate the damage froma collision.

[0006] There has also been proposed a system that provides extensivemonitoring of the vehicle and traffic situation in order to prioritizepresentation of information to the driver. The goal of this system is tomanage the stream of information to the driver while taking account ofthe driving task, conditions, and the physical, perceptual and cognitivecapacities of the driver. The support provided is designed to improvethe driver's focus and to re-focus the attention of a distracted driveras s/he undertakes navigation, maneuvering and control of the vehicle.The overall system architecture incorporates an analyst/planner thataccepts inputs from sensors, includes a stored repertoire of drivingsituations, and records information about the driver. Additionally, thesystem includes a dialogue controller for managing communication withthe driver. The system also monitors the driver and integrates thedriver's condition into the decisions made by the warning and controlsystems.

[0007] None of the existing systems undertake the monitoring of a rangeof sensor data, nor do they provide for evaluation of the driver'scognitive load. Such systems additionally fail to consider the driver'sactivity in the cockpit that is not directly related to the driving tasksuch as opening and closing windows, tuning the radio, etc. For example,existing systems either do not monitor the driver at all, or monitor thedriver relative to static “model” behavior as opposed to actual dynamicdriver performance and/or habits. Thus, these systems do not provideinformation in synchronization with the driving task, nor do theyattempt to minimize distractions and/or to redirect a distracteddriver's attention to critical events.

[0008] Additionally, previous systems that have attempted to assessdriver performance have been limited to lane-following capability, thatis, evaluating how well the driver maintains the position of the vehiclerelative to the edges of the lane in order to generate a parameterrepresenting the driver's lane-following ability. The parameter isperiodically determined, and if it falls below an established level, awarning, such as a buzzer or visual indication, is presented to thedriver. This system is limited in that it only provides lane-followingevaluation and does not account for deliberate lane departures such asto avoid a hazard, is not integrated to receive a spectrum of sensorinput, and does not include driver condition and driver activity data.Though such a measure will identify degraded vehicle control, it isquestionable whether it will identify cognitive or mental distraction.

[0009] Furthermore, none of these systems provide feedback to the driverrelative to their overall performance, nor do they provide feedbackrelative to improving driver performance.

[0010] Thus, there is a need to provide information to the vehicleoperator in synchronization with the driving task so as to improveoperator focus, minimize distractions and ensure the operator's abilityto assimilate and use the information. There is a further need tore-direct a distracted operator's attention from non-mission criticalactivities to prioritized information and/or tasks necessary to maintainsafe operation of the vehicle. There is an additional need to providefeedback to the vehicle operator relating to performance and to provideadditional feedback designed to assist the operator in improvingperformance.

BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The invention is described in terms of several preferredembodiments with reference to the attached figures wherein likereference numerals refer to like elements throughout.

[0012]FIG. 1 is a block diagram of an operator performance assessmentsystem in accordance with a preferred embodiment of the invention.

[0013]FIG. 2 is block diagram illustrating an interface of the driverwith the vehicle in accordance with a preferred embodiment of theinvention.

[0014]FIG. 3 is a block diagram illustration of a wireless communicationdevice according to a preferred embodiment of the invention.

[0015]FIG. 4 is a flow chart illustrating the steps of a method ofassessing vehicle operator performance in accordance with a preferredembodiment of the invention.

[0016]FIG. 5 is a flow chart illustrating the steps of a method ofimproving vehicle operator performance in accordance with a preferredembodiment of the invention.

[0017]FIG. 6 is a flow chart illustrating the steps of a method ofsynthesizing a response to vehicle operating conditions in accordancewith a preferred embodiment of the invention.

[0018]FIG. 7 is a flow chart illustrating the steps of a method ofproviding feedback to a vehicle operator in accordance with a preferredembodiment of the invention.

[0019]FIG. 8 is a flow chart illustrating the steps of a method ofconfiguring a service state of a wireless communication device inaccordance with a preferred embodiment of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] A system is adapted to assess information incoming to a vehicleoperator, to prioritize the information based upon a number ofconditions relating to the vehicle operation, the operating environment,the activity of the operator and the physical condition of the operator,and to provide to the operator the most pertinent information for thegiven set of conditions. As used throughout this specification, theterms vehicle operator and driver are used interchangeably and each areused to refer to the person operating the vehicle in the manner in whichthe vehicle is intended to be operated.

[0021] In another embodiment of the invention, the system monitorsvarious data sources, including the vehicle operation, the operatingenvironment, the activity of the operator and the condition of theoperator, and provides an assessment of the operator's performance. Indoing so, the system may additionally identify the particular vehicleoperator such that the assessment may be made relative to operatorpreferences, past driving performance and habits.

[0022] The system is further adaptable to assist the operator inimproving performance. The system monitors various data sources,including the vehicle operation, the operating environment, the activityand condition of the operator, over a period of operation and recordsthe operator's performance. The performance may be compared withaccepted good practices, and a report may be provided to the operatorindicating how the operator's performance compares with the acceptedgood practices and/or with the operator's previous driving performanceand/or habitual behavior. The system may record operator performanceover a number of periods of operation, and provide comparisons ofoperator performance from period to period.

[0023] The system is further adaptable to act in response to anassessment of the vehicle operation, the operating environment, theactivity of the operator and the condition of the operator to avoid ormitigate a problem situation associated with operation of the vehicle.

[0024] Referring then to FIG. 1, a system 100 includes a sensor fusionmodule 102, a response selector module 104 and an action generator 106.The sensor fusion module 102, response selector module 104 and actiongenerator 106 are illustrated in FIG. 1 as separate elements forpurposes of clarity and discussion. It will be appreciated these modulesmay be an integrated into single module. Moreover, it will beappreciated that each of these modules, or an integrated module, mayinclude a suitable processing device, such as a microprocessor, digitalsignal processor, etc., one or more memory devices including suitablyconfigured data structures, and interfaces to couple the system 100 tovarious vehicle sensors and to interface with a driver 108.

[0025] The sensor fusion module 102 receives data from numerous sourceswithin and surrounding the vehicle. As illustrated in FIG. 1, the sensorfusion module 102 receives vehicle operating data 112, vehicleenvironment data 114, driver condition data 116 and driver activity data118.

[0026] The vehicle operating data 112 encompasses data produced by thevarious vehicle sensors. Vehicle condition monitoring sensors arepervasive in an automobile. These sensors monitor numerous parameterssuch as engine operating parameters, vehicle speed, transmission andwheel speed, vehicle acceleration in three axes, chassis function,emission control function, etc. These sensors may also provide datarelated to vehicle diagnostics.

[0027] Vehicle environment data 114 encompasses data related to theenvironment in which the vehicle is operating, e.g., the roadconditions, traffic conditions, weather, etc. The vehicle environmentdata 114 may be provided by sensors that also provide vehicle-operatingdata 112. For example, road surface and traction estimates may beprovided by anti-lock braking, traction control and chassis controlsystem sensors. Vehicle location may be provided by an on-boardnavigation system utilizing global positioning system (GPS) technology,or location information may be provided by a wireless communicationdevice (e.g., a cellular telephone) and associated wirelesscommunication network. Radar, laser, ultra-sonic and video systems canprovide a map of objects near the vehicle and their motion relative tothe vehicle. Weather and time of day may also be monitored directly orderived from reported sources.

[0028] Driver condition data 116 and driver activity data 118 may beprovided by various cockpit monitoring systems. Seat sensors and/orinfrared sensors may sense the number and locations of passengers in thevehicle. Floor and steering wheel sensors may indicate the position ofthe driver's feet and hands. Video or imaging sensors may monitor head,body, hand and feet movements of the driver, and the operative statesand driver usage of infotainment and telematics systems may also bemonitored.

[0029] As will be appreciated, numerous sources of data exist within andabout the vehicle environment, which may be utilized by the system 100.Several data types have been described above, others will be describedin connection with the operation of the system 100, and still others notspecifically referred to herein may be used without departing from thescope and spirit of the invention. It will be appreciated that as newtechnologies introduce new types and sources of data and new types andsources of information into the vehicle, the system 100 may be adaptedto utilize these additional sources of data to manage how the existingand new sources of information are presented to the driver.

[0030] In other words, the system 100 will monitor anything of atechnical nature that the driver might be touching or using in thecockpit of the vehicle so that the system 100 knows as much as possibleabout what the driver is doing at any given moment. Further, the use ofvideo or imaging technology, seat sensors and microphones in the cockpitallows the system 100 to determine the location and position of thedriver, the noise level, and the presence of passengers and otherpotential sources of distractions. The radar, laser, video and infra-redsensors deployed around the perimeter of the vehicle monitor traffic andweather conditions, obstacles, lane markings, etc. The drivers' presentcondition and driving performance is inferred from direct measures, suchas video, and from comparison of current performance with pastperformance and known good performance practices.

[0031] In addition to obtaining data from a variety of sources, thesystem 100 interfaces with the vehicle operator/driver 108. Whileoperating the vehicle, the driver 108 is engaged in a number ofdifferent actions, such as, but certainly without limitation, applyingthe accelerator or brakes, turning the steering wheel, checking blindspots, adjusting the radio, receiving a cellular telephone call,obtaining navigation information, carrying on a conversation with apassenger, quieting the kids in the rear seat, etc. Each of the driver'sactions, which for discussion purposes are illustrated as box 110 inFIG. 1, are fed back to the sensor fusion module 102 via the sensors.Additionally, as will be described in greater detail, the system 100presents information, actions and tasks to the driver 108 via the actiongenerator 106. This “closed” loop operation may continue for a givensituation until the situation is resolved. In one very limited exampleto illustrate the concept, a change oil soon indication may be generatedby the powertrain management system on the vehicle. Previously, thisindication would cause a “service engine” or “change engine oil” lightto be illuminated on the vehicle instrument panel as soon as thepowertrain management system generated the indication. The lightsuddenly appearing among the instruments may temporarily distract thedriver. If at the time the light is illuminated the driver isnegotiating traffic or otherwise in a situation requiring full attentionto the driving task, the distraction may present a hazard. In accordancewith the preferred embodiments of the invention, the non-critical datarelating to changing the engine oil may be saved until conditions allowfor the information to be presented to the driver at a time less likelyto create a hazard situation. In that regard, the system 100 operatescontinuously taking in data and re-timing its presentation to thedriver. Moreover, the system 100 continuously evaluates the informationto be provided to the driver to determine when and how to best provideit to the driver. This operation of the system 100 may be illustrated byan additional example. A low fuel alert may initially be a non-criticalpiece of information relative to current driving conditions but maybecome a critical piece of information if the driver is about to passthe last gas station, as system 100 is informed by the on-boardnavigation system, within the remaining range of the vehicle.

[0032] Referring to FIG. 2, a number of interfaces exists between thedriver 108 and the vehicle and hence to the system 100. Variousinterfaces are discussed below, and may include driver identification200, instrumentation and alerts 202, vehicle controls 204, drivercondition sensors 206 and driver activity sensors 208.

[0033] Due to the wide variation in human skill-levels, physical size,and personal preferences and tastes, there are many situations where itwould be useful for the system 100 to “recognize” who is attempting toenter and/or drive the vehicle. In that regard, the driveridentification interface 200 may be configured as a personal portableuser interface (PPUI). A PPUI may exist in many forms, but in essencecaptures preference, performance and habit data associated with aparticular driver. The PPUI may be encoded on a smart card or embeddedin the vehicle to be activated by a fingerprint reader, voicerecognition system, optical recognition system or other such means.

[0034] In various embodiments, the PPUI may function as a securitysystem granting or limiting access to the vehicle or the vehicle'signition system, and bars access to unauthorized persons or disables thevehicle when an unauthorized person attempts to drive the vehicle. ThePPUI may also capture driver preferences as it relates to a number ofactive safety features. Through the PPUI (driver identificationinterface 200), the system 100 is informed of the driver preferences.For example, the driver may select what types, under what conditions andhow alerts are communicated. For example, a driver may prefer to receivean alert each time the system 100 detects too short a headway relativeto a speed of travel. For another driver, a high level of alert might beperceived as a nuisance resulting in the alerts being ignored and/or thesystem 100 being disabled. Similarly, a driver may wish to haveimmediate access to all in-coming cell phone calls, while another drivermay wish to have only certain calls put through. The PPUI as part of thedriver identification interface 200 permits each operator of the vehicleto establish choices ahead of time.

[0035] The PPUI may also function in a driver performance improvementand/or driving restriction enforcement tool. The PPUI may be used tomonitor driving performance and report to a traffic enforcementauthority. This would allow a habitual traffic offender to retaindriving privileges in a court-monitored fashion. Driving performance maybe recorded for subsequent review, and a method of improving driverperformance is described herein. Additionally, the PPUI may be used toimplement controls on the usage of the vehicle. For example, a parentmay restrict the distances and locations a vehicle may be taken or thehours of the day the vehicle may be operated by a newly licensed driver.An employer may monitor the driving habits of its fleet drivers.

[0036] In operation, the system 100 is programmed to recognize, based onthe received data, “situations” and “conditions” that might arise duringoperation of a vehicle. The system 100 may be configured to actuate,relative to priorities for the presentation of information and thethresholds for the levels of alerts, warnings and alarms. The driveridentification interface 200 including the PPUI provides the driver withchoices relating to the priorities, thresholds and interfaces, andoperates to synchronize the choices with the appropriate driver.

[0037] The instrumentation and alerts interface 202 is used by thesystem 100 to inform, advise and in the appropriate situations alert andwarn the driver 108. The instrumentation and alerts interface 202 mayinclude visual, audio, haptic or other suitable indicators. Visualindicators may include gages, lighted indicators, graphic andalphanumeric displays. These visual indicators may be located centrallywithin the instrument panel of the vehicle, distributed about thevehicle, configured in a heads-up-display, integrated with rearview andside view mirrors, or otherwise arranged to advantageously convey theinformation to the driver 108. The audio indicators may be buzzers oralarms, voice or other audible alerts. The haptic alerts may includeusing the chassis control system to provide simulated rumble stripes,pedal or steering wheel feedback pressure, seat movements and the like.The actuation of any one or more of the indicators or alerts iscontrolled by the system 100 in order to synchronize the timing ofinformation as it is provided to the driver.

[0038] The vehicle controls interface 204 includes the primary controlsused by the driver to operate the vehicle. These controls include thesteering wheel, accelerator pedal, brake pedal, clutch (if equipped),gear selector, etc. These controls may include suitable position and/oractuation sensors and may further include at least in the case of theaccelerator pedal, brake pedal and steering wheel rate of input and/orforce of input sensors. Additional sensor data may include yaw rate ofthe vehicle, wheel speed indicating vehicle speed and traction, tirepressure, windshield wiper activation and speed, front and/or rearwindow defogger activation, audio system volume control, and seat beltusage sensors.

[0039] The driver condition interface 206 utilizes various sensors toinfer driver condition. For example, an alert driver continuously makessteering corrections to maintain the vehicle in its lane. By monitoringsteering wheel sensors, the system 100 gathers data about the frequencyand amplitude of the corrections to infer if the driver has becomeimpaired. Speed sensors may also be queried in a similar manner. Videoor other imaging sensors provide direct measurement of the drivers'condition via monitoring of such criteria as driver blink rate and gaze.

[0040] The driver activity interface 208 utilizes various sensors andimaging technology to determine the activity of the driver. That is, todetermine if the driver, in addition to operating the vehicle, isadjusting the radio or heating, ventilation and air conditioning (HVAC)controls, initiating or receiving a wireless communication, receivingnavigation information, and the like. These sensors may include seatpressure sensors to determine the number of passengers in the vehicle,and the activities of the passengers and video or other imagingtechnology to observe the driver's movements.

[0041] Referring again to FIG. 1, and as previously noted, the sensorfusion module 102 receives all of the various sensor inputs, includingthose measuring vehicle condition, driver condition, driver activity andoperating environment (e.g., weather, road and traffic conditions), andproduces a set of conditions or master condition list. The conditionsrepresent the current discrete state of each thing the sensor fusionmodule 102 is monitoring. For example, the speed condition may be in oneof the following states at any point in time: “stopped,” “slow,”“normal,” “fast,” and “speeding.” The states are determined based uponlearned thresholds between the states and based on history and knowngood practices. The sensor fusion module 102, given the master conditionlist, evaluates the current drivers tasks and activities, such as tuningthe radio, listening to e-mail or other potentially distracting tasks,to produce an estimated driver cognitive load. The cognitive load ofeach static task may be determined externally by controlled experimentswith a set of test subjects (e.g., tuning the radio might use 15.4percent of a driver's attention). The total cognitive load is theweighted sum of each of the individual tasks. The weighting may be fixedor may change, for example exponentially, given the number of concurrenttasks.

[0042] The master condition list and the estimated driver cognitive loadis then provided to the response selector module 104. The responseselector module looks at the conditions, current driving situation andcognitive load to determine if a problem exists and further assesses theseverity of the problem. The response selector module 104 further takesinto account driver preferences, to choose a response appropriate to thedriver's present task and prioritizes the presentation of alerts,warnings and other information to the driver. The response selectormodule 104 may incorporate a reflex agent that uses decision tree orlook-up tables to match states with desired actions. Alternatively, anadaptive, i.e., learning, goal-seeking agent may be used. Thus, theresponse selector module 104 synthesizes and summarizes sensor datecreating a correct response to any given condition change.

[0043] In one possible implementation, the response selector module 104may include programmer-entered parameters which are used to determine ifa condition change a) creates a problem, b) solves a problem, c)escalates a problem, d) initiates a driver task, e) initiates an agenttask, f) completes a driver or agent task, g) changes the situation orh) is innocuous. The estimated cognitive load may be used to determinean urgency of an identified problem or whether a response to the problemshould be initiated by the driver or by an agent. For example, anincoming cellular phone call may be directed to the driver if thedriver's estimated cognitive load is below a threshold value forreceiving cellular telephone calls. If the driver's cognitive loadexceeds the threshold value for receiving cellular telephone calls, thenthe cellular telephone call may be forwarded to voice mail (i.e., anagent device).

[0044] The response selector 104 activates the action generator 106 inorder to effect the selected response. The action generator 106 may be alibrary of actions that the system is equipped to perform, such as inthe above example, forwarding a cellular telephone call to voice mail.The library may include actions along with instructions, which may besoftware instructions for causing the associated processor to act, i.e.,to actuate all potential alerts and warnings that can potentially beprovided to the driver.

[0045] Fusion of sensor data, including data relating to the driver'scondition and activity allows the system 100 to operate to assess driverperformance. As noted, the system 100 is operable to identify a driverthrough the driver identification interface 200. During operation of thevehicle by the driver, the system 100 monitors several aspects of driverperformance to arrive at a driver performance assessment value.

[0046] In one embodiment, the system 100 may monitor the driver's lanefollowing ability. Information on lane-exceedence is recorded relativeto the use of turn signals and to subsequent movement of the vehicle todetermine whether the lane change was intentional or unintentional.Additionally, the system 100 may monitor gaze direction, blink rates,glance frequency and duration to determine the driver's visual scanningbehavior including the use of mirrors and “head checks” when changinglanes. The information may be used in comparison to known “good habits”to assess performance, and at the same time, may be used to develop ametric reflecting the driver's normal patterns, which can be used as abaseline to compare changes in driving behavior as well as to monitordegradation or improvement in driving skill.

[0047] Additional information that may be taken into consideration toassess driver performance includes application of the accelerator andbrakes. The driver's use of the accelerator and brakes is recorded andgiven a numeric value. Again, using comparison algorithms to known “goodhabits” and to past performance an assessment of how smoothly the driveris braking and/or accelerating may be made as well as the number andseverity of panic stops. Accelerator and brake pedal data may also beused in conjunction with metrics of headway maintenance, as monitored bythe system 100. Doing so allows the system 100 to determine whether thedriver is waiting too long to brake relative to obstacles in the forwardpath of the vehicle and even to determine whether the driver is prone tounsafe headway when vehicle speed control devices are used.

[0048] In addition to assessing driver performance, the system 100 maybe adapted to assist in the improvement of driver performance.Communication of the driver assessment to the driver encourages thedriver to perform better. The system 100 may also provide specificadvice relating to improving driver performance. For example, themonitoring of driver performance may extend temporally (recording andcomparing the driver's performance over time) and spatially (consideringperformance variation on familiar, frequently-traveled routes) toinclude all of the times that a particular driver has driven theequipped vehicle. The driver assessment, i.e., driver performance,including alerts, warnings and suggestions for improved performance, isthen provided to the instrumentation/alerts module 202 for communicationto the driver. A library of pre-recorded messages to the driver may beaccessed by the system 100 and appropriate messages constituting reportsand suggestions, are chosen. For example, the system 100 may havedetected that the driver has not been doing head-checks before changinglanes, and may draw the driver's attention to that fact and state thereason that merely glancing at the mirror is not a good substitute for ahead-check. Additional messages may include reminders about improvingfuel economy or specifically identify an area of driving performancethat deteriorated over the course of trip.

[0049] Communication of performance improvement information may be madereal time; however, to avoid creating further distractions for thedriver, the information may be stored and communicated to the driverfollowing a driving activity. Triggering events and/or thresholds may beused to actuate delivery of the performance improvement messages.Alternatively, the driver may optionally select to activate theinterface. The stored performance information may also be downloadedfrom the vehicle and used as part of a classroom or simulator-basedcontinuing training program, a driver skills assessment program or atraffic enforcement program.

[0050] To encourage usage of the system 100 to improve drivingperformance, the feedback may be configured to appeal to particularcategories of drivers. For example, for younger drivers, voice andlikenesses of motor racing personalities may be used to convey theinformation, while for other drivers well known and trustedpersonalities may be used.

[0051] One particularly example of potential driver distraction relatesto usage of cellular telephones. As described, the system 100synthesizes and prioritizes all incoming information, including cellulartelephone calls. For example, the system 100 may provide two potentialcut-offs of cellular telephone calls to a driver without completelyprohibiting calls. In the first instance, the caller is informed, by apre-recorded message, that the call is being completed to a personpresently driving a vehicle. The caller is then given the option ofhaving the call sent directly to voice mail or putting the call throughto the driver. Before the call is completed to the driver, the system100 evaluates the situation, conditions and the driver's cognitive loadto determine if the response, sending the call through, is appropriate.If the system 100 determines that the potential for driver distractionis beyond certain desired limits, e.g., the required driver cognitiveload will exceed a threshold, the incoming call may be held and/orautomatically transferred to voice mail with an appropriate pre-recordedmessage.

[0052] The system 100 may be configured to substantially limit thenumber of calls coming in to the driver. Many times a caller does notknow the person they are calling is driving, and if they did, may nothave called. As described above, the system 100 provides a mechanism forinforming the caller that they are calling a driver and provides theoption to divert the call to voice mail. Alternatively, the system 100may be configured to give the driver the option of accepting callstransparent to the caller. In such an arrangement the incoming call isidentified to the driver via a hands-free voice interface. The drivermay then accept the call, refer the call to voice mail, refer the callto a forwarding number or to terminate the call, all of which may beaccomplished without the caller's knowledge. Alternatively, the callcompletion may be delayed shortly, with an appropriate message beingprovided to the caller. The system 100 may then complete the call afterthe short delay once it is determined that the driver's cognitive loadis at an acceptable level.

[0053] The system 100 may also be adapted to take “corrective” action inthe event that an on-going call is coupled with a degradation of drivingperformance. If after accepting a cellular telephone call the system 100determines that the driver's cognitive load has increased beyond athreshold level and/or if there is a degradation in driving performancebelow a threshold level, the system 100 may automatically suspend thecellular telephone call. In such instance, a message is provided thatinforms the caller they are being temporarily placed on hold. The system100 may also offer the caller an option to leave a voice mail message.Additionally, so that the driver is aware of the call interruption, anappropriate message is provided to the driver indicating that the callhas been placed on hold. The driver likewise may refer the caller tovoice mail.

[0054] As with other aspects of the operation of the system 100, thedriver's preferences as to cellular telephone usage is provided to thesystem 100 via the driver identification interface 200. The system 100may also operate with other wireless communication devices includingpersonal digital assistants (PDAs) and pagers for receiving email andtext and data messages.

[0055] To take advantage of the ability of the system 100 to prioritizeincoming cellular telephone calls with other information presented tothe driver requires the cellular telephone be communicatively coupled tothe system 100 and controllable by the system 100. A stand-alonecellular telephone, that may not be adaptable to the system 100, may beadapted to operate in a context aware manner.

[0056]FIG. 3 illustrates a hand-held cellular telephone 300 including aprocessor 302, a memory 304, a sensor fusion module 306 and a pluralityof sensors, one of which is illustrated as sensor 308. While shown asseparate elements, it will be appreciated that these elements of thecellular telephone 300 can be integrated into a single unit or module.Alternatively, a sensor fusion module including appropriate processingcapability may be provided as an add-on device to existing cellulartelephones. The sensor(s) 308 may take in such data as ambient lighting,temperature, motion and speed, date and time and location. Of course,where the cellular telephone 300 is operated in a wireless communicationnetwork environment information such as location, speed, date and timemay be provide by the network. However, the sensor 308 may be a GPSdevice capable of determining location, speed, time and day using theGPS satellite system.

[0057] The sensor fusion module 306 receives the data from the varioussensors and creates a master condition list that is communicated to theprocessor 302 controlling the operation of the cellular telephone. Theprocessor 302 operates in accordance with a control program stored inthe memory 304 and using the master condition list to provide contextaware operation of the cellular telephone 300. Context aware operationof the cellular telephone 300 can be illustrated by the followingexamples.

[0058] In one example of context aware operation, the cellular telephoneis determined to be moving at a speed of 60 kilometers per hour (kph).This condition is reported by the sensor fusion module 306 as part ofthe master conditions list to the processor 302. The processor 302infers from this speed condition that the cellular telephone is with adriver of a vehicle, and thus enters a service state where incomingcalls are screened. One form of screen is as described above, whereinthe caller is first advised that they are calling a driver and offeredthe option of leaving a voice message.

[0059] In another example of context aware operation, the cellulartelephone is determined to be at approximately human body temperature.This condition is reported by the sensor fusion module 306 as part ofthe master conditions to the processor 302. The processor 302 operatesin accordance with the control program and using the master conditionlist determines the cellular telephone 300 is likely located close tothe user's body. Instead of operating in a ringing service state, thecellular telephone 300 is caused to operate in a vibrate service stateto announce an incoming call.

[0060] The following Table I set forth various sensor mechanism, contextestimates and operation service states. TABLE I Context Aware WirelessDevice Service States Sensor Mechanism Context Estimates Service StateNetwork Infrastructure In an airplane No use in an airplane Capabilityof In a moving (to comply with FAA determining location vehicleregulations-subject & velocity through Near or on a override)triangulation or highway Limit when driving Doppler shift Not in amoving In intersections analysis vehicle In cities Capability of Carriedby On highways recording and someone walking Above certain comparing orrunning speeds temporally and Time of day Limit by location spatiallythe location Day of week Theaters, and velocity (above) concerts, housesto enable tracking of worship, etc. Map matching Limit by time of day,database determines day of week location relative to Hours of rest,infrastructure, meals, meetings, buildings, points of family time, etc.interest Workday rules Clock/timer vs. Weekend rules Specify callersStandalone Handset/ All of the above plus: If close, vibrate insteadAdd-on Module How close to of ringing Location normal human In no lightand sound Velocity body muffled, ring louder Track temperature? If inpocket but not next Light (Photometer) How likely in a to body, ringinstead of Temperature purse or vibrating (Thermometer) briefcase? (darkVibrate instead of Acceleration and sound ringing (vibration andmuffled) Modulate volume of orientation) In a pocket? (low ringer andspeaker (Accelerometer) light level, not relative to ambient Backgroundnoise body noise level (Microphone) temperature) Ringing alarm Chemicalsensor On a belt? Smoke sensor Determine level Clock of ambient noiseAltimeter Determine presence of chemical/smoke In-vehicle All of theabove, plus: 1. If driver's glance All vehicle control 1. Gaze trackerdoes not return to and accessory knows that driver windshield or sensorsis looking mirrors within TBD All externally elsewhere than seconds,deployed sensors through the communication determining windshield or atdiverted to voice information about the the mirrors mail. drivingenvironment 2. Navigation 2. Message delayed or All actuators and systemknows diverted to voice cockpit sensors to that a mail. determine whatthe complicated 3. Message delayed or driver is doing maneuver or adiverted to voice All sensors deployed dangerous curve mail. todetermine the is coming up. 4. Message taken driver's condition 3.Vehicle sensors 5. Call interrupted, indicates caller offered voicepotentially mail or hold, person dangerous called informed thatdegradation of call is being the vehicle dropped and will be 4. Driveralready resumed when safe engaged with a to do so. different wirelessdevice 5. Potentially dangerous situation arises while driver alreadyengaged in wireless communication

[0061] In accordance with the preferred embodiments of the invention,methods of: assessing vehicle operator performance, informing thevehicle operator to improve the operator's performance, responsesynthesis in a driver assistant system, improving driver performancethrough performance feedback and context aware device operation aredescribed in connection with FIGS. 4-8.

[0062] Referring to FIG. 4, a method 400 of assessing vehicle operatorperformance begins at step 402 with receiving vehicle operating datafrom the vehicle relating to the vehicle operating condition. Step 402involves receiving at the sensor fusion module 102 data from the varioussensors, systems and device in the vehicle data relating operation ofthe vehicle. This data may include vehicle speed and vehicleacceleration, throttle application, brake application, steering wheelinput, throttle position, rate of change of throttle position,additional available throttle input and throttle applicator pressure,brake position, rate of change of brake position, additional availablebrake input and brake applicator pressure, steering wheel position, rateof change of the steering wheel, operator pressure applied to thesteering wheel, additional available steering input and other operatingparameter of the vehicle such as oil temp, oil pressure, coolant temp,tire pressure, brake fluid temp, brake fluid pressure, transmissiontemp., misfire, windshield wiper activation, front/rear defoggerapplication, diagnostic systems, etc.

[0063] At step 404, an interior portion of the vehicle is monitored toprovide data to the sensor fusion module 102 relating to activities ofthe driver. Monitored activities may include monitoring the usage ofvehicle system controls by the driver, such as driving controls,telematics systems, infotainment systems, occupant comfort controlsincluding HVAC, seat position, steering wheel position, pedal position,window position, sun visors, sun/moon roof and window shades andcommunication controls. Monitoring activities may also includemonitoring activities of the vehicle passengers.

[0064] At step 406, the vehicle environment external to the vehicle ismonitored to provide data to the sensor fusion module 102 relating tothe operating environment of the vehicle. The operating environment datamay include road condition, lane following, headway data, trafficcontrol data and traffic condition data.

[0065] At step 408, the vehicle operator is monitored to provided datato the fusion module 102 relating to the condition of the driver. Thedriver physical condition may include fatigue or intoxication or apsychological condition of the driver. Additionally, a distraction levelof the driver may be monitored.

[0066] At step 410 the driver performance is assessed. The driver'sperformance may be assessed by inferring driver performance from thevehicle operating data, the operator activity data, the environment dataand the operator condition data. Such an inference may be drawn usinginference engine or a rules-based decision engine. Alternatively, fuzzylogic or adaptive, goal-seeking may be used.

[0067] Referring to FIG. 5, a method 500 of informing a driver toimprove driver performance begins at step 502 with receiving vehicleoperating data from the vehicle relating to the vehicle operatingcondition. Step 502 involves receiving at the sensor fusion module 102data from the various sensors, systems and device in the vehicle datarelating operation of the vehicle. This data may include vehicle speedand vehicle acceleration, throttle application, brake application,steering wheel input, throttle position, rate of change of throttleposition, additional available throttle input and throttle applicatorpressure, brake position, rate of change of brake position, additionalavailable brake input and brake applicator pressure, steering wheelposition, rate of change of the steering wheel, operator pressureapplied to the steering wheel, additional available steering input andother operating parameter of the vehicle such as oil temp, oil pressure,coolant temp, tire pressure, brake fluid temp, brake fluid pressure,transmission temp., misfire, windshield wiper activation, front/reardefogger application, diagnostic systems, etc.

[0068] At step 504, an interior portion of the vehicle is monitored toprovide data to the sensor fusion module 102 relating to activities ofthe driver. Monitored activities may include monitoring the usage ofvehicle system controls by the driver, such as driving controls,telematics systems, infotainment systems, occupant comfort controlsincluding HVAC, seat position, steering wheel position, pedal position,window position, sun visors, sun/moon roof and window shades andcommunication controls. Monitoring activities may also includemonitoring activities of the vehicle passengers.

[0069] At step 506, the vehicle environment external to the vehicle ismonitored to provide data to the sensor fusion module 102 relating tothe operating environment of the vehicle. The operating environment datamay include road condition, lane following, headway data, trafficcontrol data and traffic condition data.

[0070] At step 508, the vehicle operator is monitored to provided datato the fusion module 102 relating to the condition of the driver. Thedriver physical condition may include fatigue or intoxication or apsychological condition of the driver. Additionally, a distraction levelof the driver may be monitored.

[0071] At step 510, the driver's cognitive load is estimated. Thedriver's cognitive load may take into account driver preferences, pastperformance and habits. Then, at step 512, vehicle information isprioritized based upon the driver's cognitive load for communication tothe driver.

[0072] Referring to FIG. 6, a method 600 of synthesizing a response toan operating situation of a vehicle begins at step 602 with thegeneration of a master condition list. The master condition list isgenerated by the sensor fusion module 102 and is a fusion of the varioussensor data available within the vehicle. The sensor data may be anyavailable data within the vehicle including: vehicle operating data,driver activity data, environment data, driver condition data, driverpreference data, driver action feedback data.

[0073] From the master condition list, at step 604, an operatingsituation is determined. The operating condition may be: the existenceof a problem condition, the existence of a problem correction, theexistence of a problem escalation, the existence of an operator taskrequirement, the existence of an agent task requirement, the existenceof a completion of an operator task, the existence of a completion of anagent task. Additionally, at step 606, an operator cognitive load isdetermined.

[0074] At step 608, a response to the operating situation is determinedbased on the operator cognitive load. The response may be synchronizingan information flow to the driver, generating an alert to the driver,providing an alert including audio alerts, a visual alerts and hapticalerts, suspending or terminating operation of selected services withinthe vehicle.

[0075] Referring to FIG. 7, a method 700 of improving driver performancethrough performance feedback begins at step 702 with receiving vehicleoperating data from the vehicle relating to the vehicle operatingcondition. Step 702 involves receiving at the sensor fusion module 102data from the various sensors, systems and devices in the vehicle anddata relating operation of the vehicle. This data may include vehiclespeed and vehicle acceleration, throttle application, brake application,steering wheel input, throttle position, rate of change of throttleposition, additional available throttle input and throttle applicatorpressure, brake position, rate of change of brake position, additionalavailable brake input and brake applicator pressure, steering wheelposition, rate of change of the steering wheel, operator pressureapplied to the steering wheel, additional available steering input andother operating parameter of the vehicle such as oil temp, oil pressure,coolant temp, tire pressure, brake fluid temp, brake fluid pressure,transmission temp., misfire, windshield wiper activation, front/reardefogger application, diagnostic systems, etc.

[0076] At step 704, an interior portion of the vehicle is monitored toprovide data to the sensor fusion module 102 relating to activities ofthe driver. Monitored activities may include monitoring the usage ofvehicle system controls by the driver, such as driving controls,telematics systems, infotainment systems, occupant comfort controlsincluding HVAC, seat position, steering wheel position, pedal position,window position, sun visors, sun/moon roof and window shades andcommunication controls. Monitoring activities may also includemonitoring activities of the vehicle passengers.

[0077] At step 706, the vehicle environment external to the vehicle ismonitored to provide data to the sensor fusion module 102 relating tothe operating environment of the vehicle. The operating environment datamay include road condition, lane following, headway data, trafficcontrol data and traffic condition data.

[0078] At step 708, the vehicle operator is monitored to provided datato the fusion module 102 relating to the condition of the driver. Thedriver's physical condition may include fatigue or intoxication or apsychological condition of the driver. Additionally, a distraction levelof the driver may be monitored.

[0079] At step 710, the driver's performance assessment is determinedand is recorded so that at step 712, the driver's performance assessmentmay be reported to the driver. Step 712 includes reporting the driverperformance assessment upon conclusion of vehicle operation or reportingthe operator performance assessment during operation of the vehicle.Moreover, the driver's performance assessment may be recorded for afirst period of vehicle operation and for a second period of vehicleoperation and include a comparison of the two performances.

[0080] The method may further include the step of receiving driverpreference data and recording the driver performance assessment based onthe driver preference data. Additionally, the driver performanceassessment may include a score for each of a plurality of aspects ofvehicle operation. Reporting the driver performance assessment may be byvisual indication, audio indication or haptic indication.

[0081] Referring to FIG. 8, a method 800 for configuring a service stateof a wireless communication device begins at step 802 with receiving aset of device operating parameters defining a preferred service state ofthe wireless communication device for a device operator. At step 804,context data is received from at least one context data source. Thedevice operating parameters includes at least a context parameter. Thecontext parameter and the context data may each relate to: a speed ofthe wireless communication device, a location of the wirelesscommunication device, time, an activity of the device operator, acognitive load of the device operator, an operation of a vehicleincluding vehicle operating data and environment data, ambient lighting,altitude and ambient sound. The data received may be a fusion of datafrom a variety of sources, such as from within the vehicle where thewireless communication device is communicatively coupled to the vehicle.The device operating parameters may be provided via a personal portableuser interface, such as vehicle identification interface 200.

[0082] At step 806, a service state of the wireless communication deviceis set. The service state may be: call forwarding, call forwarding tovoice mail, voice activated, ringing mode, call completion delay andcalling party identification, etc.

[0083] The wireless communication device may be a cellular telephone, apager, a personal digital assistant or other computing device includingpersonal computers and web browsers.

[0084] The invention has been described in terms of several preferredembodiments, and particularly to systems and methods for synthesizingand summarizing information and the presentation of information to adriver. Modifications and alternative embodiments of the invention willbe apparent to those skilled in the art in view of the foregoingdescription. This description is to be construed as illustrative only,and is for the purpose of teaching those skilled in the art the bestmode of carrying out the invention. The details of the structure andmethod may be varied substantially without departing from the spirit ofthe invention, and the exclusive use of all modifications, which comewithin the scope of the appended claims is reserved.

We claim:
 1. A method of synthesizing a response to an operatingsituation of a vehicle; the method comprising the steps of: generating amaster condition list, the master condition list being a fusion ofsensor data within the vehicle; determining the existence of anoperating situation based on the master condition list; determining adriver cognitive load based upon the master condition list; anddetermining a response to the operating situation in view of theoperator cognitive load and the operating situation.
 2. The method ofclaim 1, wherein the sensor data comprises vehicle operating data. 3.The method of claim 1, wherein the sensor data comprises operatoractivity data.
 4. The method of claim 1, wherein the sensor datacomprises environment data.
 5. The method of claim 1, wherein the sensordata comprises operator condition data.
 6. The method of claim 1,wherein the sensor data comprises operator preference data.
 7. Themethod of claim 1, wherein the sensor data comprises driver actionfeedback data.
 8. The method of claim 1, wherein the step of determiningthe existence of an operating situation comprises determining theexistence of a problem condition.
 9. The method of claim 1, wherein thestep of determining the existence of an operating situation comprisesdetermining the existence of a problem correction.
 10. The method ofclaim 1, wherein the step of determining the existence of an operatingsituation comprises determining the existence of a problem exacerbation.11. The method of claim 1, wherein the step of determining the existenceof an operating situation comprises determining the existence of anoperator task requirement.
 12. The method of claim 1, wherein the stepof determining the existence of an operating situation comprisesdetermining the existence of an agent task requirement.
 13. The methodof claim 1, wherein the step of determining the existence of anoperating situation comprises determining the existence of a completionof an operator task.
 14. The method of claim 1, wherein the step ofdetermining the existence of an operating situation comprisesdetermining the existence of a completion of an agent task.
 15. Themethod of claim 1, wherein the step of determining a response to theoperating situation comprises synchronizing an information flow to anoperator of the vehicle.
 16. The method of claim 1, wherein the step ofdetermining a response to the operating situation comprises generatingan alert to the driver.
 17. The method of claim 1, wherein the alertcomprises one of an audio alert, a visual alert and a haptic alert. 18.The method of claim 1, wherein the step of determining a response to theoperating situation comprises terminating operation of selected serviceswithin the vehicle.